{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enviroment: \n",
    "Open AI gym [MountainCar-v0](https://github.com/openai/gym/wiki/MountainCar-v0)\n",
    "\n",
    "### Observation\n",
    "Type: Box(2)\n",
    "\n",
    "| Num  | Observation | Min   | Max  |\n",
    "| ---- | ----------- | ----- | ---- |\n",
    "| 0    | position    | -1.2  | 0.6  |\n",
    "| 1    | velocity    | -0.07 | 0.07 |\n",
    "\n",
    "### Actions\n",
    "\n",
    "Type: Discrete(3)\n",
    "\n",
    "| Num  | Action     |\n",
    "| ---- | ---------- |\n",
    "| 0    | push left  |\n",
    "| 1    | no push    |\n",
    "| 2    | push right |\n",
    "\n",
    "### Reward\n",
    "\n",
    "-1 for each time step, until the goal position of 0.5 is reached. As with MountainCarContinuous v0, there is no penalty for climbing the left hill, which upon reached acts as a wall.\n",
    "\n",
    "### Starting State\n",
    "\n",
    "Random position from -0.6 to -0.4 with no velocity.\n",
    "\n",
    "### Episode Termination\n",
    "\n",
    "The episode ends when you reach 0.5 position, or if 200 iterations are reached.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. gym enviroment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.5518721,  0.       ])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "env = gym.make(\"MountainCar-v0\")\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Q Table and Eligibility Trace setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "LEARNING_RATE = 0.5\n",
    "DISCOUNT = 0.95\n",
    "EPISODES = 10000\n",
    "SHOW_EVERY = 200\n",
    "Q_TABLE_LEN = 20\n",
    "LAMBDA = 0.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "DISCRETE_OS_SIZE = [Q_TABLE_LEN] * len(env.observation_space.high)\n",
    "discrete_os_win_size = (env.observation_space.high - env.observation_space.low) / DISCRETE_OS_SIZE\n",
    "\n",
    "# q_table = np.random.uniform(low=0, high=1,\n",
    "#                             size=(DISCRETE_OS_SIZE + [env.action_space.n]))\n",
    "\n",
    "q_table = np.zeros((DISCRETE_OS_SIZE + [env.action_space.n]))\n",
    "e_trace = np.zeros((DISCRETE_OS_SIZE + [env.action_space.n]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 20, 3)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_table.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decay epsilon "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "epsilon = 1  # not a constant, qoing to be decayed\n",
    "START_EPSILON_DECAYING = 1\n",
    "END_EPSILON_DECAYING = EPISODES//2\n",
    "epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Help functions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_discrete_state (state):\n",
    "    discrete_state = (state - env.observation_space.low) // discrete_os_win_size\n",
    "    return tuple(discrete_state.astype(int))\n",
    "\n",
    "def take_epilon_greedy_action(state, epsilon):\n",
    "    discrete_state = get_discrete_state(state)\n",
    "    if np.random.random() < epsilon:\n",
    "        action = np.random.randint(0,env.action_space.n)\n",
    "    else:\n",
    "        action = np.argmax(q_table[discrete_state])\n",
    "    return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Rewards Recorder setup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "ep_rewards = []\n",
    "aggr_ep_rewards = {'ep':[],'avg':[],'min':[],'max':[]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Train the Agent "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode: 0\n",
      "episode: 200\n",
      "episode: 400\n",
      "episode: 600\n",
      "episode: 800\n",
      "episode: 1000\n",
      "episode: 1200\n",
      "episode: 1400\n",
      "episode: 1600\n",
      "episode: 1800\n",
      "episode: 2000\n",
      "episode: 2200\n",
      "episode: 2400\n",
      "episode: 2600\n",
      "episode: 2800\n",
      "episode: 3000\n",
      "episode: 3200\n",
      "episode: 3400\n",
      "episode: 3600\n",
      "episode: 3800\n",
      "episode: 4000\n",
      "episode: 4200\n",
      "episode: 4400\n",
      "episode: 4600\n",
      "episode: 4800\n",
      "episode: 5000\n",
      "episode: 5200\n",
      "episode: 5400\n",
      "episode: 5600\n",
      "episode: 5800\n",
      "episode: 6000\n",
      "episode: 6200\n",
      "episode: 6400\n",
      "episode: 6600\n",
      "episode: 6800\n",
      "episode: 7000\n",
      "episode: 7200\n",
      "episode: 7400\n",
      "episode: 7600\n",
      "episode: 7800\n",
      "episode: 8000\n",
      "episode: 8200\n",
      "episode: 8400\n",
      "episode: 8600\n",
      "episode: 8800\n",
      "episode: 9000\n",
      "episode: 9200\n",
      "episode: 9400\n",
      "episode: 9600\n",
      "episode: 9800\n"
     ]
    }
   ],
   "source": [
    "for episode in range(EPISODES):\n",
    "    # initiate reward every episode\n",
    "    ep_reward = 0\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        print(\"episode: {}\".format(episode))\n",
    "        render = True\n",
    "    else:\n",
    "        render = False\n",
    "\n",
    "    state = env.reset()\n",
    "    action = take_epilon_greedy_action(state, epsilon)\n",
    "    \n",
    "#   reset e_trace IMPORTANT\n",
    "    e_trace = np.zeros(DISCRETE_OS_SIZE + [env.action_space.n])\n",
    "    \n",
    "    done = False\n",
    "    while not done:\n",
    "\n",
    "        next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "        ep_reward += reward\n",
    "        \n",
    "        next_action = take_epilon_greedy_action(next_state, epsilon)\n",
    "\n",
    "        # if render:\n",
    "        #     env.render()\n",
    "\n",
    "        if not done:\n",
    "            \n",
    "            delta = reward + DISCOUNT * q_table[get_discrete_state(next_state)][next_action] - q_table[get_discrete_state(state)][action]\n",
    "\n",
    "            e_trace[get_discrete_state(state)][action] += 1\n",
    "            \n",
    "            q_table += LEARNING_RATE * delta * e_trace\n",
    "            \n",
    "            e_trace = DISCOUNT * LAMBDA * e_trace\n",
    "            \n",
    "        elif next_state[0] >= 0.5:\n",
    "#             print(\"I made it on episode: {} Reward: {}\".format(episode,reward))\n",
    "            q_table[get_discrete_state(state)][action] = 0\n",
    "\n",
    "\n",
    "        state = next_state\n",
    "        action = next_action\n",
    "\n",
    "    # Decaying is being done every episode if episode number is within decaying range\n",
    "    if END_EPSILON_DECAYING >= episode >= START_EPSILON_DECAYING:\n",
    "        epsilon -= epsilon_decay_value\n",
    "\n",
    "    # recoard aggrated rewards on each epsoide\n",
    "    ep_rewards.append(ep_reward)\n",
    "\n",
    "    # every SHOW_EVERY calculate average rewords\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        avg_reward = sum(ep_rewards[-SHOW_EVERY:]) / len(ep_rewards[-SHOW_EVERY:])\n",
    "        aggr_ep_rewards['ep'].append(episode)\n",
    "        aggr_ep_rewards['avg'].append(avg_reward)\n",
    "        aggr_ep_rewards['min'].append(min(ep_rewards[-SHOW_EVERY:]))\n",
    "        aggr_ep_rewards['max'].append(max(ep_rewards[-SHOW_EVERY:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Rewards')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEKCAYAAAA1qaOTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8lNXVwPHfyb6ThT0JBBAEREAIiGIVBQV3pa611WqtdWlftVoVfVvtZl+1q9paaWutVakrSl0AcUNUluDCDgkQICQQspE9k+W+fzzPhBCyTJLZ53w/n/nM5Flm7jMDc+Zu54oxBqWUUqqvwnxdAKWUUsFBA4pSSim30ICilFLKLTSgKKWUcgsNKEoppdxCA4pSSim30ICilFLKLTSgKKWUcgsNKEoppdwiwtcF8LT+/fubrKwsXxdDKaUCxvr160uMMQN6el7QB5SsrCxycnJ8XQyllAoYIrKnN+dpk5dSSim30ICilFLKLTSgKKWUcoug70PpSGNjIwUFBdTX1/u6KB4RExNDRkYGkZGRvi6KUiqEhGRAKSgoIDExkaysLETE18VxK2MMpaWlFBQUMGLECF8XRykVQkKyyau+vp60tLSgCyYAIkJaWlrQ1r6UUv4rJAMKEJTBxCmYr00p5b9CNqAo5S8+L/ycNUVrfF0MpfosJPtQlPIHeyv38ui6R/m44GMSoxL5+IqPiQzXgRQqcGkNRSkvq22s5U9f/IlL3ryEdQfWccHIC6hyVPFZ4We+LppSfaI1FB+55JJL2LdvH/X19dx+++00Nzeze/duHn30UQCeffZZ1q9fzxNPPMEvf/lLXnjhBTIzM+nfvz9Tp07l7rvv9vEVqJ4yxrA0fym/zfktxbXFXDjyQu6Yegcp0SmsLFjJ0vylnJF5hq+LqVSvhXxA+fl/N7OlsNKtzzl+aBIPXnhCl8c888wzpKamUldXx7Rp03j//feZOXNma0B56aWXeOCBB8jJyeG1117jyy+/pKmpiSlTpjB16lS3lld5xz0r72Fp/lLGpY7jt2f8lpMGntS6b87wOSzLX0ZDcwPR4dE+LKVSvadNXj7y+OOPM2nSJGbMmMG+ffvYvXs3I0eOZPXq1ZSWlrJ9+3ZmzpzJqlWruPjii4mNjSUxMZELL7zQ10VXvbCvah9L85fy7XHfZtH5i44KJgBzs+ZS01jDqoJVPiqhUn0X8jWU7moSnvDRRx+xYsUKPv/8c+Li4pg1axb19fVceeWVvPzyy4wdO5ZLL70UEcEY4/XyKfdbnr8cgO+M/w7hYeHH7J8+eDqpMakszV/K7OGzvV08pdxCayg+cPjwYVJSUoiLi2Pbtm2sXr0agPnz5/PGG2+waNEirrzySgBOO+00/vvf/1JfX091dTVvv/22L4uuemlZ/jImDpjI0IShHe6PCItgzrA5fFzwMbWNtV4unVLuoQHFB+bNm0dTUxMTJ07kpz/9KTNmzAAgJSWF8ePHs2fPHqZPnw7AtGnTuOiii5g0aRLz588nOzubfv36+bL4qof2VO5ha9lW5g6f2+Vxc7PmUtdUx8r9K71UMqXcK+SbvHwhOjqad999t8N9b7311jHb7r77bh566CFqa2s5/fTTueuuuzxdROVGS3cvBeCcrHO6PG7qoKn0j+3P8vzlzMua542iKeVWGlACwE033cSWLVuor6/nuuuuY8qUKb4ukuqBpflLmTJwCoPjB3d5XHhYOGcPP5vXc1+nprGG+Mh4L5VQKffQgBIAXnzxRV8XQfXSzoqd5FXksWD6ApeOn5c1j0XbFvHRvo84f+T5Hi6dd3xW+BmxEbHHjGxTwccnfSgicrmIbBaRFhHJbrM9S0TqROQr+/bXNvumishGEckTkcdFMyCqALA0fylhEtZtc5fT5IGTGRg3kKX5Sz1cMu9YU7SG21bcxu9yfufroigv8FWn/CZgPtBR7+NOY8xk+3Zzm+1PATcBo+2bNjIrv2aMYenupWQPyqZ/bH+XzgmTMOZmzeXT/Z9S6XDvhFtv2314N3d+dCdNpomCqgJfF0d5gU8CijFmqzFmu6vHi8gQIMkY87mxJmY8B1zisQIq5QY7yneQX5nP3KyuR3e1Ny9rHo0tjXy490MPlczzKuoruO3924gMi+Sbo79JaX0pdU11vi6W8jB/HDY8QkS+FJGPReQb9rZ0oO1PnAJ7m1J+a1n+MsIlnDnD5/TovBP7n8jQ+KEB2+zlaHZw+4e3c7DmIH86809MH2wNgS+sLvRxyZSneSygiMgKEdnUwe3iLk4rAoYZY04Cfgy8KCJJQEf9JZ1OIReRm0QkR0RyDh061LcL8ZElS5bwf//3f74uhuolZyJI5wz4nhAR5mbNZXXhairqKzxUQs8wxvDzz3/OF8Vf8MuZv2TywMlkJGYAaLNXCPBYQDHGzDHGTOjg9mYX5zQYY0rtx+uBncAYrBpJRptDM4BOf+4YYxYaY7KNMdkDBgxwzwV52UUXXcR9993n62KoXtpStoV9VfuYN6J3XX1zR8ylyTTx/t733Vwyz/r7xr+zZOcSbp18K+eNPA+A9ASrMaGgWgNKsPOrYcMiMgAoM8Y0i8hIrM73XcaYMhGpEpEZwBrgWuAJX5a1L/Lz85k3bx6nnXYaq1evZtKkSVx//fU8+OCDFBcX88ILL7BlyxZycnJ48skn+e53v0tSUhI5OTkcOHCARx99lMsuu8zXl6G6sGz3MiIkgtnDepeXa3zqeDITM1mav5Rvjvmmy+ftr97f41rNoPhBLg8aANh1eBd1jcf2h2wp28LjXz7O+SPP5+aJR8bTpMakEhsRqzWUEOCTgCIil2IFhAHA2yLylTFmLnA68AsRaQKagZuNMWX2abcAzwKxwLv2re/evQ8ObHTLU7UafCKc23VzVV5eHq+88goLFy5k2rRpvPjii6xatYolS5bw8MMPc8klR485KCoqYtWqVWzbto2LLrpIA4ofM8awLH8Zpww9hX7RvUuTIyKcN+I8nt7wND/79GfcPuV20mLTOj2+tK6Ux798nMW5izGdtwZ3KDIskutOuI7vn/h94iLjOj1ub+VeHln3CCsLOk8Nc9LAk/j5qT+n7ah+ESE9IZ391ft7VC4VeHwSUIwxi4HFHWx/DXitk3NygAkeLprXjBgxghNPPBGAE044gdmzZyMinHjiieTn5x9z/CWXXEJYWBjjx4/n4MGDXi6t6okNJRsorCnktpNu69Pz3HjijTiaHfx7679ZsWcFt06+lSvHXklk2JFlghtbGnlp20v85au/UNdUx7Xjr2Xa4Gkuv4bBsDx/eWtT1V1T7+LcEeceFRBqG2v528a/8a/N/yIyLJLbp9zO6OTRxzxXmISRPTi7w/VcMhIytMkrBPhVk5dPdFOT8JTo6CP/6cLCwlr/DgsLo6mpqcvjNaW9f1u6eymRYZGcmXlmn54nJiKGH2f/mEtHX8ojax/hkXWP8Frua9w3/T5OHnIya4vW8pu1vyGvIo+ZQ2dyz/R7GNlvZI9fZ1bmLK44/goeXvMw935yLy/veJkF0xcwJmUM7+5+l9+t/x3FtcVcNOoi7phyBwPiet4vmZGYwdoDazHGoHOSg5cGFKXcqMW0sHzPck5LP43EqES3POeIfiN4as5TfLjvQx5d9yg3Lr+R41OOZ3v5dtIT0vnTmX/izMwz+/RFPXngZBadv4jX817n8S8e54q3rmBkv5HkVeQxLnUcvzvjd0weOLnXz5+ekE5tUy0VDRWkxKT0+nmUf/PHeShKBaz8ynyKa4v7XDtpT0Q4a9hZvHnJm/xw8g9paG7gtsm38cbFb3DWsLPc8qs/PCycy8dczluXvsUVY67AGMPPTvkZi85f1KdgAujQ4RAhwd58kp2dbXJyco7atnXrVsaNG+ejEnlHKFyjP3pvz3v8+KMf858L/sMJad5fDdRf5ZbnMn/JfB47/bFeD6VW3iMi640x2d0feTStoSjlRnnleQjSq76MYKZzUUKDBhSl3CivIo+MxAxiI2J9XRS/EhcZR2pMqjZ5BTkNKEq5UV5FHsclH+frYvglHToc/DSgKOUmjmYHeyv3akDpRHpiOvurdHJjMNOAopSb5Ffm02SaNKB0IiMhg6KaIppajp1npYKDBhSl3GRnxU4ARiWP8nFJ/FN6QjrNppmDtZrpIVhpQFHKTXLLcwmXcEb0G+Hrovgl51wUbfYKXhpQlHKTnRU7GZY0jKjwKF8XxS/p0OHgpwHFB/Lz8xk7diw33ngjEyZM4JprrmHFihXMnDmT0aNHs3btWtauXcupp57KSSedxKmnnsr27daKyb///e+54YYbANi4cSMTJkygtrbWl5ejbDsP79T+ky4Mjh9MuITr0OEgFvK5vB5Z+wjbyra59TnHpo7l3un3dnlMd+nrn3vuOVauXElERAQrVqzg/vvv57XXXuOOO+5g1qxZLF68mF//+tc8/fTTxMV1nnJceUd9Uz17K/dy3ojzfF0UvxURFsHg+MFaQwliIR9QfKW79PWHDx/muuuuIzc3FxGhsbERsLIRP/vss0ycOJEf/OAHzJw505eXoWy7D+/GYLRDvhsZiRm6LkoQC/mA0l1NwlO6S1//05/+lDPPPJPFixeTn5/PrFmzWo/Pzc0lISGBwsJOV0FWXpZXkQegTV7dyEjI4MN9H/q6GMpDtA/FTx0+fJj0dKsT89lnnz1q++23387KlSspLS3l1Vdf9VEJVVt5FXlEhEUwLGmYr4vi19IT0imrL6O2Ufv9gpEGFD91zz33sGDBAmbOnElzc3Pr9jvvvJNbb72VMWPG8I9//IP77ruP4uJiH5ZUgRVQspKyjlpNUR3LOXS4sFpr18Eo5Ju8fCErK4tNmza1/t22BtJ2344dO1q3//KXvwTgmWeead2WmZlJXl6eh0urXLGzYicT+0/0dTH8Xtuhw8elaPNgsNEailJ9VNtYy/7q/doh74LWyY3aMR+UNKAo1UfOlCv6i7t7KdEpxEbE6lyUIBWyASWYV6oM5mvzRzrCy3UiQnpCus5FCVIhGVBiYmIoLS0Nyi9eYwylpaXExMT4uighI68ij+jwaDISMnxdlICQkZihNZQgFZKd8hkZGRQUFHDo0CFfF8UjYmJiyMjQLzdv2Vmxk5H9RhIeFu7rogSEjIQM1hStwRiDiPi6OMqNQjKgREZGMmKEZoRV7pFbkcv0wdN9XYyAkZGYQV1THeUN5aTGpPq6OMqNQrLJSyl3qXRUUlxbrP0nPdA6dFibvYKOBhSl+qB1hJcGFJc5+5p06HDw0YCiVBeW5y/nhmU3UN9U3+H+1hFeOmTYZUMThgJaQwlGPgkoInK5iGwWkRYRyW63b6KIfG7v3ygiMfb2qfbfeSLyuGhvnvKCzwo/Y92BdTy/9fkO9+eV5xEbEcuQ+CFeLlngiouMIzUmVWsoQchXNZRNwHxgZduNIhIBPA/cbIw5AZgFNNq7nwJuAkbbt3neKqwKXc4vvb9t+BsldSXH7N9ZsZNR/UYRJlrZ7wkdOhycfPK/wBiz1RizvYNd5wAbjDFf28eVGmOaRWQIkGSM+dxYk0eeAy7xYpFViCqqKWJ82ngczQ6e/PLJY/bnVuRqc1cv6OTG4ORvP6vGAEZElonIFyJyj709HWj7r6/A3tYhEblJRHJEJCdY55ooz2sxLRRVFzF98HSuGnsVr+e+zvayI7+DyuvLKasv0w75XshIyOBAzQGaWpp8XRTlRh4LKCKyQkQ2dXC7uIvTIoDTgGvs+0tFZDbQUX9Jp9PcjTELjTHZxpjsAQMG9Ok6VOgqqy/D0eJgSPwQbp50M0nRSTy27rHWDAuacqX3MhIzaDbNHKg54OuiKDfyWEAxxswxxkzo4PZmF6cVAB8bY0qMMbXAO8AUe3vbqd8ZgC6ooDzKuWbH0ISh9Ivuxy2TbmHNgTV8XPAxcCSgaJbhnnPORdGO+eDibzPllwH3iEgc4ADOAP5gjCkSkSoRmQGsAa4FnvBhOVUIKKyxAopzBNcVx1/BS9tf4nc5v2Pm0JnsrNhJYmQig+IG+bKYAcmZxr6gqoCTh5zs0jkHag7w4tYXqWqs8mTRgkZCZAJ3Zd/l1df0SUARkUuxAsIA4G0R+coYM9cYUy4ivwfWYTVpvWOMeds+7RbgWSAWeNe+KeUxRdVFwJF5E5FhkdydfTe3vX8bL21/idzyXEYlj9J8VL0wKG4Q4RLuUg3F0ezguS3PsXDDQhpbGkmOTvZCCQNfSkwKdxECAcUYsxhY3Mm+57GGDrffngNM8HDRlGpVWF1IYmQiiVGJrdu+kf4NZgyZwVNfP4UxhnOyzvFhCQNXRFgEQ+KHdDt0+ON9H/PIukfYV7WPszLP4ifTftJau1H+x9+avJTyG0U1RQxJOHrCoojwk2k/4fL/Xk6LaWF0ymgflS7wpSemd1pD2VO5h0fWPsIn+z8hKymLp+c8zanpp3q5hKqnNKAo1YnCmkLS448dnT4mZQzzR8/n1R2vaod8H2QkZPB67uuc+uKxgaK2qZbo8GjumnoX14y7hsjwSB+UUPWUBhSlOlFUXUT2oOwO990x5Q6GJQ5j6qCpXi5V8Pj2uG8TGxHb4b64yDiuOv4qBsTpsP9AogFFqQ5UOiqpbqxmaPzQDvf3i+7H9ROu93KpgstxKcdx7/R7fV0M5Ub+NlNeKb/gHOHVvg9FKdU5DShKdaB1UmMnNRSl1LE0oCjVgdZJjVpDUcplGlCU6kBRdRHR4dGkxaT5uihKBQwNKEp1oLCmkCHxQ3QWvFI9oAFFqQ4UVRfpKoxK9ZAGFKU6UFhT2JrDSynlGg0oSrVT31RPWX2Z1lCU6iENKEq1U1RzdJZhpZRrNKAo1U7rpEatoSjVIxpQlGrHOQdFayhK9YwGFKXaKawuJFzCGRg30NdFUSqgaEBRqp2imiIGxg0kIkxzpyrVExpQlGqnsLpQ+0+U6gUNKEq1U1RTpP0nSvWCBhSl2mhqaaK4tlhrKEr1ggYUpdoori2m2TRrDaULjc0tvi6C8lMaUJRqQ9dB6dq/V+/hhAeX8fA7W2nSwKLa0YCiVBvOWfKhsA7KoaoGbnl+Pef+6RO+2lfR5bHNLYZfvbWFn76xiYzkWBau3MV1/1xLWY3DS6VVgcClgCIio0Qk2n48S0T+R0SSPVs0pbzPWUMJ9j6UdzcWMfePK3l/WzGl1Q3M/8unPLZsGw1NzcccW+to4ubn1/P3Vbv57qlZvPfjM3j0somsyy/nwidWsbHgsA+uQPkjV2sorwHNInIc8A9gBPCix0qllI8U1RSRGpNKTESMr4viEYdrG7nzpa+45YUvGJocw9s/Oo0Vd53BN6dk8OcPd3Lxk5+yufBIgDhYWc+VT6/m/a0HeejC8Tx00QmEhwlXZGfy6s2nYIzhm3/9jFfXF/jwqpS/cDWgtBhjmoBLgT8aY+4EgvsnnApJhdWFQdt/snLHIeb+cSVLvi7k9tmjWXzrTEYPSiQpJpLHLp/EP67LprTGwcVPfsrj7+eyaf9hLv3zp+w8VM3frs3muzNHHPV8EzOS+e+PTiN7eAp3v/I1P3tzE44m7VcJZa5OBW4UkauB64AL7W2RnimSUr5TVFPEmJQxvi6GW9Q3NrO58DBf7TvM2t2lLNt8kOMGJrDw2qlMzDi2xXr2uEEsvyOFny3ZzO/f28Hv39vBoKRoXv7BKUxI79fha6QlRPPcDdN5dNl2Fq7cxZB+sdwya5SnL035KVcDyvXAzcCvjTG7RWQE8HxvX1RELgceAsYB040xOfb2a4CftDl0IjDFGPOViEwFngVigXeA240xprdlUKo9YwxFNUXMypzl66L0StHhOj7JLeHLvRV8va+C7QeraG6x/osMTorhB2eM5M45Y4iJDO/0OVLio3ji6pOYd8Jglm85wH3njmVIv9guXzciPIz7zxvH0k0H2FpU6dZrUoHFpYBijNkC/E+bv3cD/9eH190EzAeebvc6LwAvAIjIicCbxpiv7N1PATcBq7ECyjzg3T6UQamjlNaX0tDc4Dcd8qXVDbyzsYgBidGkJ8eRkRJLclxk6zr39Y3NrN5VysodJXySe4jc4moAkmIimJSZzC1jRzExox+TMpMZlNSzPqHzJw7h/Ik9ex+Gpcaxt6y2R+eo4NJlQBGRjUCntQBjzMTevKgxZqv9/F0ddjWwyD5uCJBkjPnc/vs54BI0oCg3cq6D4i+TGl/K2cejS7cftS0+Kpz0lFiSYiLZsP8wjqYWoiLCOHlEKldkZ3L6mAGMGZTQ3f8tj8hMjWP55gNef13lP7qroVxg399m3//bvr8G8PRPkSuBi+3H6UDbYSQF9jal3Ma5Doq/1FBKqhzERYXz8g9OoaC8jv0VdRSU11JQXkdZjYPvzBjO6WMGMD0rldiozpuxvGVYahylNQ6qG5pIiNZMzaGoy0/dGLMHQERmGmNmttl1n4h8Cvyis3NFZAUwuINdDxhj3uzqdUXkZKDWGLPJuamj4nVx/k1YzWMMGzasq5dSqpW/1VAqah2kxEUxIb1fp53i/mRYahwA+8pqGTckycelUb7g6s+IeBE5zRizCkBETgXiuzrBGDOnD+W6Cru5y1YAZLT5OwMo7OK1FwILAbKzs7XjXrmksKaQxMhEEqMSfV0UAMpqHaTGR/m6GC5zBpS9GlBClqsB5QbgnyLSD6tmcNje5nYiEgZcDpzu3GaMKRKRKhGZAawBrgWe8MTrq9BVVF3kVylXymsbSY4LnNH5bWsoKjR1G1DsL/jjjDGTRCQJEGNMn3ItiMilWAFhAPC2iHxljJlr7z4dKDDG7Gp32i0cGTb8Ltohr9yssMa/JjWW1zjISovzdTFc1i8ukqSYCB3pFcK6DSjGmBYR+SHwsjHGLYPMjTGLgcWd7PsImNHB9hxggjteX6mOFFUXMXXQVF8Xo1W53YcSSIal+cfQYWMMxkCLMVTUNbKvrJZ95XXsK6uloLyWfWV1NLcYzp84hAsnDqVfFzXBhqZmPt5+iA+3H2JyZj8unpze5Vye3vosr4QfLfqS4wcnMuv4Acw6fiCjB/ZuxF5zi6GgvJbhaV32TLidq01e74nI3cBLQI1zozGmzCOlUsrLqhxVVDVW+U0NpbG5har6psALKKlxbDtQ5bHnL6txsPNQNfklNewprSW/9Mh9raMZYwwt3fSapsVHkZEaR21DE//7xiZ+8dYWzhk/iMumZvCN0QMIDxOaWwxrdpXy5leFvLupiMr6JqIjwli0di+/eXcbV08fxndmDGdocteTPl2VV1zNzc+vJzkuipLqBh5+ZxsPv7ON9ORYzjh+ALPGDGDmcf2J72b03OG6Rl7J2ce/Ps+nqdmw8p4ziQz3XlL5nvShwJHhw2D1pYx0b3GU8o3WLMN+0odSXmulhU+ND5w+FLDmoqzYUkxLiyEszH1zYfKKq3jygzyWfF3YGjDCw4TMlFiGp8Vz0rBkEmMiCBNBRBCwH0NCdASZqXEMS7Umhzq/lI0xbNpfyavr9/Hm14W8taGIgYnRzDyuP5/mlVBc1UB8VDhzTxjMhZOHctpx/cnJL+dfn+Xz9Mc7WbhyF3NPGMR3Tx3B1OEptBhDS5uaUYuB2Mhwwrt5H8pqHHzvX+uIDA/jhRtPJjM1jv0VdXy8/RAfbS/mzS/38+KavUSFhzF9RGpr7WXUgPjW2ktecRXPfpbPa+v3U9fYzPSsVL47M4swL89HcnWm/Ijuj1IqcDnXQfGXGkpFbSMAyQFYQ3E0t3Cwqr7blC2u2FpUyZMf5PHOpiJiI8P53mkjmHlcf7LS4klPie3Tr28R4cSMfpyY0Y/7zx/Hh9uKeXX9flZsPcgpI9O4aPJQZo8ddNQcn1NGpXHKqDT2ldXy/Oo9LFq7l3c2dj6ZMz05lt9dMYkZI9M63N/Q1MzN/15P0eF6Fn1/Bpn2wIb05Fi+dfIwvnXyMBxNLeTsKbOb3Yr51dtb+dXbW8lMjWXWmIHkl9bwSW4JURFhXDxpKNedmuWzYeYuzz4SkQnAeKA1h4Mx5jlPFEopb/O3Gopz4apAGjYMbYYOl9b2KaBs2n+Yx9/PZfmWgyRER3DrrFF877SRHns/oiPCmTdhCPMmuPb5Z6bGseC8cdw+ZzRvbSiiqKKeMIGwMCFMBGelZNHavVz9t9XccsYo7pgzhqiIIwHQGMP9r29ibX4Zf7pqMlOHp3T4WlERYZw6qj+njurPgvPGUVBey0d27eXV9QUkxUZw9zljuHr6MNISovv8XvSFSwFFRB4EZmEFlHeAc4FVgAYUFRSKaoqIDo8mLabjX5LeVmE3eQXSsGE4ei7KyZ38Ku/O0k1F3Pz8FyTFRHDHnNFcf+qILjvNfSkuKoIrsjM73f/tGcP5xX+38JePdrIqr4Q/XjmZkQMSAPjLRzt57YsC7pgzmosnu574IyMljm/PGM63ZwynsbmFcBG3Ni/2has1lMuAScCXxpjrRWQQ8HfPFUsp7yqsLmRI/BCf5MDqSFmN1eQVaDWUocmxhEnv56I0Nrfwm3e3MXZwIq/cfAqJMf4ZSFwVHx3BI5dN5MyxA7jv9Y2c//gqfnbheJJjI3ls2XYumjSU22eP7vXze7PD3RWuBpQ6e/hwkz0XpRjtkFdBpLS+lP6x/X1djFbOTvlAG+UVGR7G0OTYXg8dfjlnH3tKa3nmu9kBH0zamjdhCJMzU7jrla9Y8PpGRGDKsGQevWyi3/yIcQdXA0qOvYb834D1QDWw1mOlUsrLKh2VZCZ03nThbeU1DmIjwz0y38HTepvGvs7RzJ9W5JI9PIUzjx/ogZL51uB+Mfz7hpN55tPdfLT9EH+8anJAfr5dcXWU1632w7+KyFKsVPIbPFcspbyrylHlNzm8wEq7EmjNXU7DUuNYsbW4x+c9+1k+xVUN/PmaKUH1q72tsDDhxm+M5MZvBGcDj6ud8s8BnwCfGGO2ebZISnmf/wUUR8B1yDtlpsZRUt1AraOJuCjXGkEO1zby1Ed5nDV2INOyUj1cQuUprvboPAsMAZ4QkZ0i8pqI3O65YinlPU0tTdQ01pAU5T8ZcstqAivTcFtHkkTWuXzOX1fupLK+ibvPOd5TxVJe4FJAMcZ8APwa+CnW6K5srGSNSgUqX9NGAAAgAElEQVS8mkYrm1BStP8ElIpaR8BNanRqO3TYFcWV9fzz091cPHko44f6z2eges7VJq/3sdY/+Ryr6WuaMabnjaRK+aFKh5Xz1J+avMpqHKQGaJNXTwPK4x/k0tRs+PHZYzxZLOUFrjZ5bQAcWNl+JwITRMQ9WdGU8rEqh5XMMDHSPwJKU3MLlfVNAVtDSY6LJDE6gr2lNd0em19Sw3/W7uPq6cO8nhlXuZ+ro7zuBBCRBOB64J9Yy/v6dp6/Um7QGlD8pIZSUReYkxqdRIRMF4cO//69HUSECz866zgvlEx5mqtNXj8EvgFMBfYAz2A1fSkV8PwuoDgnNQZoQAGr2Su3uOs09psLD7Pk60JunTWKgUkxXR6rAoOrExtjgd8D640xTR4sj1Je5wwo/jLKy5l2JSVA+1DAWmjrg+1dp7F/9tN8EqIj+MEZo7xcOuUpro7yegyIBL4DICIDRERT2qug4G+d8s5Mw4GWdqWtzNQ4HE0tFFc1dLi/pcXw4fZizhw7kH6xgRs41dFcCih2tuF7gQX2pkjgeU8VSilvqnRUEiZhxEX6x/rtwdDkNbybkV4b9h+mpNrB7LHBl2IllLk6yutS4CLs5X+NMYWAf/ycU6qPqhxVJEQmECb+kbm1zLlaYwDXULobOvzB1oOECZwxZoA3i6U8zNX/QQ5jjMFa9hcR0fF9Kmj4W9qVitpGoiPCjlopMNA409h3FlDe31bM1OEpAV0LU8dyNaC8LCJPA8ki8n1gBboeigoSVY4qv+mQh8BOu+IUFRHGkH6xHa6LcuBwPZsLKzlr7CAflEx5kqvzUH4rImcDlcDxwM+MMe95tGRKeYm/BZSKWkdAd8g7dZbG/oNtVpKN2eO0/yTYuLymvB1A3gMQkXARucYY84LHSqaUl1Q6KslKyvJ1MVqV1ThIiQ/8kU/DUq2hw+19sO0gGSmxjB6Y4INSKU/qsslLRJJEZIGIPCki54jlh8Au4ArvFFEpz/LHPpSgqKGkxXGoqoE6R3PrtvrGZlbllTB77MCgXfMklHXXh/JvrCaujcCNwHLgcuBiY8zFHi6bUl7hbwGlLEiavDKdaezLjzR7fb6zlPrGFs4ap/0nwai7Jq+RxpgTAUTk70AJMMwY03VOBaUCRFNLE7VNtX4TUJpbDIfrGoNi9FPr0OHSWsYMst7f97cdJC4qnBkjdRGtYNRdDaXR+cAY0wzs1mCigkm1oxrwn1nyh+saMYaATV3fVvu5KMYYPthazDdG9yc6InCHRKvOdRdQJolIpX2rAiY6H4tIZW9fVEQuF5HNItIiItlttkeKyL9EZKOIbBWRBW32zROR7SKSJyL39fa1lWrLmXbFX0Z5taZdCYIaSkpcJAnREa0BZduBKgoP1zNbhwsHrS6bvIwxnvoZsQmYDzzdbvvlQLQx5kQRiQO2iMgiYB/wZ+BsoABYJyJLjDFbPFQ+FSL8NtNwEPShONPYO+eiOIcLzxqrs+ODlU9yTRhjthpjtne0C4gXkQisDMcOrLkv04E8Y8wuY4wD+A+ggwJUn2liSM8alhrbWkN5f+tBJmX0Y2CipqoPVv6RvOiIV7HyhRUBe4HfGmPKgHSsWopTgb1NqT7xvxqKnbo+COahwJHJjSXVDXy5r0Jnxwc5lyc29pSIrMBa1bG9B4wxb3Zy2nSgGRgKpACf2M/T0YB108Vr3wTcBDBs2LCeFFuFGL9bCyWImrzACigNTS28klOAMTo7Pth5LKAYY+b04rRvAUuNMY1AsYh8CmRj1U4y2xyXARR28doLgYUA2dnZnQYepfwtoJTXOIiKCCMugBNDtuWci/Lvz/MZlBTNCUP9431WnuFvTV57gbPsGfnxwAxgG7AOGC0iI0QkCrgKWOLDcqogUemoJFzCiY2I9XVRACivdZAaFxU0s8idQ4cLD9dzls6OD3o+CSgicqmIFACnAG+LyDJ715+BBKxRYOuAfxpjNtjLDv8QWAZsBV42xmz2QdFVkHHOkveXL7qymkaSg2AOilN6SizOt1b7T4Kfx5q8umKMWQws7mB7NdbQ4Y7OeQd4x8NFUyGmqtG/0q5U1AZ+6vq2oiPCGZIUQ0mNg5nHpfm6OMrDfBJQlPIX/pjHa9zg4OpnmDI8hTAR4qL06ybY6SesQlplQ6VfBZSK2sagGTLs9OS3pmAt+OohLc2w4kE42MNW8JQRcP7vwFPNnVv/C4cLYMYtnnl+P6QBRYW0KkcVA+L8Y+Z2c4sJmsW12vNoH9WyB2DNUzBkMoS7GIzrK2HnB3DSNZA+1f1lMgbe+xk4ajWgKBUq/KnJq7KukRYTPHNQvGLd361gcvItcO7/uX5eXTk8Nho2ve6ZgFL0NZTtAgm3alBhwTEMvDv+NmxYKa+qaqwiMdI/Akq5PakxmDrlPSpvBbxzD4yeC3N/3bNzY1Ng1Fmw+Q1oaXF/2Ta/bt2bZqgtc//z+ykNKCpkNTY3UtdU5zc1FGdACaZhwx5TvBVeuR4GjoPL/tG7GsCE+VBZAAXr3Fs2Y2DzYoiwc5ZVH3Tv8/sxDSgqZFU1+lcer/IaK4+X1lC6UX0IXrzC+sK++j8Q3cvP7/jzIDz6SG3CXfavh4q9MNFeJV0DilLBrzXtSrR/DNMNtjxeHtFYD//5lhVUvvUfSM7s/pzOxCTBcXPc3+y1eTGER0H2Ddbf1cXue24/p53yKmT5Wx6v1rVQtIbSMWNgyQ+hYC1c/i/3dKZPmA/b34a9n0PWzL4/X0uLFVBGzYa00dY2d9RQ8ldByY6O9x03B5L9IwmuBhQVsvxvLZRGIsOF+CBJDOl2mxfDxlfgrP+FEy5xz3OOmQcRsVazlzsCSsFaqNwPcx6C6ASISuh7QNm3Fp49v/P9ycPhtrUQ6ft1ZjSgqJDVuhaKn4zycs5B8Ze8Yn6lsd6avDhoApz2Y/c9b3QCjDkHtrwJ8x6B8D5+JW563erbOf5c6++EgX0LKMbA0gWQMBi+txwioo/eX5ADL10Da5+Gmbf3/nXcRPtQVMjyt8W1ymqCK4+XW615yuroPudX7p/TccJ8qDkEe1b17XlammHLGzD67CMDBRIG9a0PZdNrsD8HZv8UUoZD4uCjb+MusIZNr/yt1a/kYxpQVMjytyav8lqHDhnuSHUxrPyd1Tw16kz3P//ocyAy3qpd9MWez6zayAnzj2zrSw2lsQ7eexAGT4RJ3+r8uHN+BY218NHDvXsdN9KAokJWlaOKCInwo7VQGrWG0pEPfw1NddYXpydExVlNVFuXQHNj759n8+sQGQdj5h7ZljCo9wHl8yeteTLzfgNhXXxVDxgD2d+D9c/CwS29ey030YCiQpa/rYVSXuMgWYcMH+3gZvjiOZh2I/Qf7bnXmTDfSsey6+Pend/cBFuWWMEkKv7I9oSBUH/Y6gPqiaqD8MkfYOwFkHVa98fPug+ik2D5A1a/i49oQFEhq9LhP5mGW1oMFXWNpHo7oBhjtf37I2OsxI/RSXDGvZ59rVGzrdfp7STH/JVQW3J0cxdYNRSAmh72o3zwS2h2wNm/cO34uFTrPdr5gZWSxkc0oKiQ5U+JIavqm2huMd7vQ3n7x/DEVKg64N3XdUXuctj1ofXrOy7Vs68VGWPNnN/6FjQ5en7+5sXWEOHRZx+9PWGwdd+TjvmiDfDl83DyDyBtlOvnTbsRUkdZQbgvTXd9oAFFhSx/Cig+SQy5fz3kPAPlu2HRVVaqdX/R3Gh9MaaOsvoHvGHCfGg4bP3K74nmRmvtk+PPg8h2/XEJA617VwO2MbDsfit55ek/6Vk5IqLgnF9CyXarP8UHNKCokFXlqPKbWfJl3p4lbwwsvR/iB8ClC6HwK1j8A89k3u2NnH9Caa7VER/hpfdk5JkQk9zzZq9dH1n9LxPmH7vP2eTlasf89nch/xM4836ITe5ZOcAKalnfgA8ftsrkZTqxUYUsv6qh1Hg5j9eWN2DfarjgjzDpSmsexvIHrLb7OQ96pwydqSu3hsCOOP3IBEFviIiy5nVsfhPWPA24OFhj6xKI7melw28vvr/1PK40eTU5YPn/Qv/jYer1PSn5ESIw92F4+nRrbkpP0/r3kQYUFbL8qYZSXmtnGvZGQGmst+Y3DDwBplxrbTvlNqtGsOr3Vrv9Sd/2fDk6s32pFVRmP+i55Xk7M/ka+PIFePeenp037cZjZ7GDtYJkXJprNZR9q6Fsp5WnrC8z9odMtD6/DS/DmQ9Yw6K9RAOKCkmOZgf1zfV+V0NJ9sZ68mv+ChV74DtvHJl1LgLn/RbK8+G/d0BKlmvDVT2hfDcg1oQ+bxt+Kty3t+ed2l0NGnB1tnzZbut+6Ek9e+2OzPm5NULMi8EENKCoEOWPs+QjwoTEaA//l6w+ZDWFdDTrPDzS+nX8j7PhpW/Dje/3bJSRu5TnQ78M7/WdtBfj5lqrq7Ply/MhLAKS0vv+mvFpfX+OXtBOeRWS/C2Pl5V2xcXEkDuWwavfs5qGeqq7WeexyfCtl0HC4PlvQsH6nr9GX5XvsTLoBgtXaygVe6xA2tcElT6kAUWFJL8LKDWNpHbX3FW6E1643FqtcOt/YdGV1t+lO117kYOb4Yt/WcNwu5p1njrCWgmxsRb+fha8cZt3F4mq2GMlQgwWiXb6le5msAdBINWAokKSvy2uVWanru9QQzWseAj+MgP2fG7VLu7dDef82vr7LzOs/Q3Vnb9A21nns+7rvkCZ0+FH6+HU/4ENL1mTHz//i+cnzDXWQVWR1YcTLBIGQXMD1Fd0fVx5fsBftwYUFZL8rYZS0VFAMQY2vgpPToNVf4AJ34Qf5cCpP7LyRZ36Q+tLf8Jl1v4ns2HDK1BZdOxt82Jr1vkZ97o+6zw60Zood+vnkDENli2Av57W+3xXrqjYZ90H+C/1o7TORemiltdQbaVuCfCaWeA21inVB/7WKV9W08jU4e0CyteL4I1bYMgkuPxZGHbysScmDoJLn4Ls6+Gdn8DrN3b+IqmjrOGtPdV/NHz7Ndj+jrXY03MXwQ3LOy5PX5XnW/cB/kv9KM7Z8tUHYcDxHR9Tsce6D/Dr9klAEZHLgYeAccB0Y0yOvT0KeBrIBlqA240xH9n7pgLPArHAO/Y+36XVVAHNn2ooxhi7htKuD2XvaisFx/c/7H5Rqczp8P0PrJnWNZ0stDT6nN6PnBKBsedD5gx4bKS11K0nAkrrF2tg/1I/iis1lHL7upOzPF4cT/JVDWUTMB8reLT1fQBjzIkiMhB4V0SmGWNagKeAm4DVWAFlHvCu94qsgkmVo4rIsEhiwn2/DndVQxNNLebYPF6ledasaVdXKAwLt2Z6e1J8GsT1h5Jczzx/eb61hK7zSzgYtK2hdCZIamY+6UMxxmw1xmzvYNd44H37mGKgAsgWkSFAkjHmc7tW8hxwidcKrIKOP62F0mnalZId0P84H5SoG/1HezagJA/3/gx5T4pJhvCorgNKxR4rW7Gnsyp7mL91yn8NXCwiESIyApgKZALpQEGb4wrsbR0SkZtEJEdEcg4d8v06y8r/+GPalZS2w4bryq2mq/5jfFSqLvQfbaVp8YRgGzIMVnDsbi6Kc8hwgAdSjwUUEVkhIps6uF3cxWnPYAWLHOCPwGdAEx1naeu0/8QYs9AYk22MyR4wYEBfLkMFKX9aXKvDGkpJnnWf5sFVCnsrbbQV7NydzdYY64s1wJt9OtTdbPkgGDIMHuxDMcbM6cU5TcCdzr9F5DMgFygHMtocmgEU9rWMKnT5Vabh2g4CirMG4Jc1FLtMJXmQOc19z1tXDg2VwTVk2Clh8JEBB+0ZY+1rnwonAPlVk5eIxIlIvP34bKDJGLPFGFMEVInIDLEava8F3vRlWVVg86caSllNB2uhlOyw8jr5Y/OPc5Z9yQ73Pm+QDJ3tUMLAzhfZqjlkZSUIguv21bDhS4EngAHA2yLylTFmLjAQWCYiLcB+4DttTruFI8OG30VHeKk+8LcaSniYkBTT5r9jSS6kjrQSNvqb5OEQFun+fpTWkU5+GET7KmEQ1JZamQbaf6atQ4YD/7p9ElCMMYuBxR1szwc6nPljz1WZ4NmSqVDhXwGlkZS4yKNHnJXk+mf/CVjJC1NHun+kVxB9sR4jYSBgoKYEkoYcvS+I5t74VZOXUt7Q0NyAo8XhF6O8ah1NrN1dxsDENvNhmpugbFfXCRx9zRNDh8vzITbV/enj/UFXSwGX2+ugBEEg1YCiQk7rLPlI39ZQjDHc8+oGdh2q5t5zxx7ZUbEHWhr9P6CU7bKCn7sE45Bhp65my5fvgfiBXl8MyxM0oKiQ4y95vJ5euYu3NhTxk7ljOWNMm+HtJX48wsup/xgr6HU2cqk3gmTobIe6mi0fRNetAUWFnNbU9dG+a1r5eMchHl26jfMnDuHmM0YevdM5eirND2fJO6W5eaRXS7OVaTgImn061FWTVxDVzDSgqJDj68SQe0pr+NGLXzBmUCKPXTbx2PQvpblWvix/TsPhTAnjrn6UqiKrxhMkX6zHiIyBmH7HNnk1N8LhgqAJpBpQVMjxZUCpaWjipufWExYmLPxONnFRHQy0LMn17+YusLIgxw9wXw0lSJIjdilhEFS3m4tyuABMS9BctwYUFXIqG6w+FG+P8jLG8JNXvya3uIonr57CsLROOmFLcv0zKWR7/cdYGZHdIZiHDDt1lM8riIYMgwYUFYKqGr1fQzHG8IcVubyz8QD3nTuW00b37/jA2jJr5T5/r6GA1cfjzhqKhEG/TPc8nz/qKJ9XkNXMdMVGFXIqHZVEhUURHR7tldfbdaia+xdvZPWuMi49KZ3vf2Nk5weX+nFSyPb6j7Fmf9eW9b2/p2IPJKX3fgGwQNBRDaV8j5ViJ6nT5OkBRQOKCjnemiXvaGph4cqdPP5BHtERYfxm/olcmZ3Z9Roszl/8/jwHxak1p1du31dvDNYsw20lDARHtbV+fHSCta0836qVubqImp/TgKJCjjcCyvo95dz/+ka2H6zi/IlDePCC8QxMcmF1yJJcK09WIPQlOANKqTsCSj4c1+ME5YHFOXS4pvhIQAmiIcOgAUWFIE8urtXSYvjV21v552e7GZIUwz+uy2b2uB4sZ1uSC2mjrHxZ/i55uLUSYV/7URrrrNFPQfTF2qHWyY3FVi40sGpmY8/3XZncLAD+1SrlXp4MKKt3l/LMp7u5alomP71gPPHRPfwvVhoAQ4adwsIhddSRxcB6q2KvdR/0TV6DrXtnx3xDtTUAI4iuW0d5qZDjySav19bvJyE6ggcvPKHnwaS50U4KGSABBazhzX2toYTCkGE4Np9XkA0ZBg0oKgRVOio9UkOpaWji3U1FXDBxCLFRvehkLd8DLU2B0SHv1H+MlS23ubH3zxHMC2u1FZcKEn5koa0gGzIMGlBUiDHGeKyGsnTTAWodzXxzakb3B3ekdYRXANVQ0kZbQdD55dgb5fkQEXukjyFYhYVb2QWcTV6tNbMsnxXJ3TSgqJBS31xPY0ujRwLKa18UMDwtjuzhKb17AucKiP6cFLK9tkOHe6s8H5KHQVfDqYNFwsAjTV7l+RCV4N8523pIA4oKKZ7K41VQXstnO0uZf1JG1/NMulKyw1oXIzbZrWXzKGfw60s/SijMQXFKGHSkhlJhX3cQBVINKCqktKaud3MfyuIv9gMwf0ofZjyX5AVW/wlYwS9+YO/Xlzcm6OZidKntbPnyPUE3EEEDigopnqihGGN4/cv9nDwilczUPqy6V7Ij8AIKWH0+vW3yqiuHhsoQqqEMtCY2trQcqaEEEQ0oKqR4YrXGL/aWs7ukpved8WDlw6orC4wcXu31P673AcXZmR9kv9Q7lTDIGsRQsh0aa4OuZqYBRYUUT9RQXl2/n9jIcM47cUjvnyQQlv3tTP8xVjCsKe35uUE4F6NLifZclH1rrPsgC6QaUFRIcXdAqW9s5q0NhcybMJiEnk5kbKt1yHAAjfBySmuT06unQrGGArBvnXWvTV5KBS53B5T3thykqr6Jb07pQ3MXWF/G4VGB+cXal6HD5XsgNhVivLvYmc8ktK+hDPNdWTxAA4oKKVWOKqLDo922FsprXxQwpF8Mp4xK69sTleRaebECMY158jAIj+7d0OEg7JjuknPyZmmuFVyi+jCIww9pQFEhxZ1pV4or61m54xCXnpROeFgf5xKU5AbmCC+wgmDaqN4tB1yeHzr9J2BNZIy0g0gg1ka7oQFFhZRKR6Xbmrve+Go/LYa+je4CKw9W+e7ADSjQu+WAW5qhYl9QfrF2SuRILSUIA6lPAoqIPCYi20Rkg4gsFpHkNvsWiEieiGwXkbltts+zt+WJyH2+KLcKfD3J41XT0MTrXxTw8DtbeWbVbpZvPsCWwkoO1zVijOG19fuZnJnMqAEJfStUeb6dFDIAR3g59R9jXUdPkkRWFkJLY2g1ecGRfpQgvG5frYfyHrDAGNMkIo8AC4B7RWQ8cBVwAjAUWCEizv9lfwbOBgqAdSKyxBizxQdlVwGsylFFSkznubaamlv4dGcpi78oYNnmg9Q1NhMRJjS1mKOOS4yOoKqhiV9dMqHvhXL+sg/EOShO/e0kkWW7YYCLgTHUhgw7OWsoQVgz80lAMcYsb/PnauAy+/HFwH+MMQ3AbhHJA6bb+/KMMbsAROQ/9rEaUFSPVDmqGJY0DGMMNY5myqodlNU6KKtp4LO8Ut78upBDVQ0kxURwyUnpzJ+SztRhKZTXOigor2N/RR0F5bUUlNdR62jmkpP6kGrFqXUOSgAOGXZquxywqwGlPETS1rfnXGgrCK/bH1ZsvAF4yX6cjhVgnArsbQD72m3v4yLWXbtk4UQc0uLJl1A+sD8Sxu/fQ977b2OMVeuIs29XCtwQFUHiwEjio8IJKxJ42zovzb5Nav+Ef3dDoaoPWs0gMf3c8GQ+4qxdvXUnvP8L186pLQUJg36ZniuXP2pt8tIaistEZAUwuINdDxhj3rSPeQBoAl5wntbB8YaO+3pMB9ucr30TcBPAsGG9G+c9gASaTXOvzlX+a3ADzDDJNKXFER0eRmREGNERYUSGh5EQHUFkuA8yvw44HkbO8v7rulNMEpxxHxza2rPzBp8I4ZGeKZO/mjAfTHNQBlKPBRRjzJyu9ovIdcAFwGzj/Klo1TzavssZQKH9uLPtHb32QmAhQHZ2dqeBpyt/u+mz3pymVOg6c4GvSxAY0kbBrOAcV+SrUV7zgHuBi4wxtW12LQGuEpFoERkBjAbWAuuA0SIyQkSisDrul3i73EoppTrnqz6UJ4Fo4D17MaLVxpibjTGbReRlrM72JuA2Y6x2JxH5IbAMCAeeMcZs9k3RlVJKdUSOtDYFp+zsbJOTk+PrYiilVMAQkfXGmOyenqcz5ZVSSrmFBhSllFJuoQFFKaWUW2hAUUop5RYaUJRSSrlF0I/yEpFDwJ5ent4fKHFjcQKFXndo0esOLa5c93BjzICePnHQB5S+EJGc3gydC3R63aFFrzu0ePK6tclLKaWUW2hAUUop5RYaULq20NcF8BG97tCi1x1aPHbd2oeilFLKLbSGopRSyi00oHRAROaJyHYRyRORgF+4QEQyReRDEdkqIptF5HZ7e6qIvCciufZ9ir1dRORx+/o3iMiUNs91nX18rr2mjd8TkXAR+VJE3rL/HiEia+xreMleEgF72YSX7OteIyJZbZ5jgb19u4jM9c2VuE5EkkXkVRHZZn/up4TC5y0id9r/xjeJyCIRiQnWz1tEnhGRYhHZ1Gab2z5jEZkqIhvtcx4XOzV8l4wxemtzw0qPvxMYCUQBXwPjfV2uPl7TEGCK/TgR2AGMBx4F7rO33wc8Yj8+D3gXawXNGcAae3sqsMu+T7Efp/j6+ly4/h8DLwJv2X+/DFxlP/4rcIv9+Fbgr/bjq4CX7Mfj7X8H0cAI+99HuK+vq5tr/hdwo/04CkgO9s8ba7nw3UBsm8/5u8H6eQOnA1OATW22ue0zxlqL6hT7nHeBc7stk6/fFH+72W/gsjZ/LwAW+Lpcbr7GN4Gzge3AEHvbEGC7/fhp4Oo2x2+3918NPN1m+1HH+eMNa3XP94GzgLfs/xwlQET7zxtrvZ1T7McR9nHS/t9A2+P88QYk2V+s0m57UH/edkDZZ385Rtif99xg/ryBrHYBxS2fsb1vW5vtRx3X2U2bvI7l/EfpVGBvCwp2tf4kYA0wyBhTBGDfD7QP6+w9CMT35o/APUCL/XcaUGGMabL/bnsNrddn7z9sHx9o1z0SOAT8027q+7uIxBPkn7cxZj/wW2AvUIT1+a0n+D/vttz1Gafbj9tv75IGlGN11E4YFEPhRCQBeA24wxhT2dWhHWwzXWz3SyJyAVBsjFnfdnMHh5pu9gXUdWP92p4CPGWMOQmowWr+6ExQXLfdX3AxVjPVUCAeOLeDQ4Pt83ZFT6+1V++BBpRjFQCZbf7OAAp9VBa3EZFIrGDygjHmdXvzQREZYu8fAhTb2zt7DwLtvZkJXCQi+cB/sJq9/ggki4hz+eu219B6ffb+fkAZgXfdBUCBMWaN/ferWAEm2D/vOcBuY8whY0wj8DpwKsH/ebflrs+4wH7cfnuXNKAcax0w2h4ZEoXVWbfEx2XqE3t0xj+ArcaY37fZtQRwjuq4Dqtvxbn9WntkyAzgsF19XgacIyIp9q/Bc+xtfskYs8AYk2GMycL6HD8wxlwDfAhcZh/W/rqd78dl9vHG3n6VPSpoBDAaq8PSLxljDgD7ROR4e9NsYAtB/nljNXXNEJE4+9+887qD+vNuxy2fsb2vSkRm2O/ltW2eq3O+7lTyxxvWiIgdWKM7HvB1edxwPadhVVc3AF/Zt/Ow2ovfB3Lt+1T7eAH+bF//RiC7zXPdAOTZt+t9fW09eA9mcWSU10isL4g84BUg2t4eY/+dZ+8f2eb8B+z3YzsujHbx9Q2YDOTYn/kbWCN4gv7zBn4ObAM2Af/GGr/yNQcAAAKaSURBVKkVlJ83sAirr6gRq0bxPXd+xkC2/T7uBJ6k3SCPjm46U14ppZRbaJOXUkopt9CAopRSyi00oCillHILDShKKaXcQgOKUkopt9CAopQLRKRZRL5qc+syC7WI3Cwi17rhdfNFpH9fn0cpb9Bhw0q5QESqjTEJPnjdfKw5AyXefm2lekprKEr1gV2DeERE1tq34+ztD4nI3fbj/xGRLfY6FP+xt6WKyBv2ttUiMtHeniYiy+2kjk/TJqeSiHzbfo2vRORpsdZ5CReRZ8Va/2OjiNzpg7dBKUADilKuim3X5HVlm32VxpjpWLOJ/9jBufcBJxljJgI329t+Dnxpb7sfeM7e/iCwylhJHZcAwwBEZBxwJTDTGDMZaAauwZoRn26MmWCMORH4pxuvWakeiej+EKUUUGd/kXdkUZv7P3SwfwPwgoi8gZUGBax0ON8EMMZ8YNdM+mEtmjTf3v62iJTbx88GpgLr7IXzYrES//0XGCkiTwBvA8t7f4lK9Y3WUJTqO9PJY6fzsfIoTQXW25ltu0oP3tFzCPAvY8xk+3a8MeYhY0w5MAn4CLgN+Hsvr0GpPtOAolTfXdnm/vO2O0QkDMg0xnyItdBXMpAArMRqskJEZgElxlqjpu32c7GSOoKV6O8yERlo70sVkeH2CLAwY8xrwE+x0tQr5RPa5KWUa2JF5Ks2fy81xjiHDkeLyBqsH2hXtzsvHHjebs4S4A/GmAoReQhrRcUNQC1HUo7/HFgkIl8AH2OlZMcYs0VE/hdYbgepRqwaSZ39PM4fhwvcd8lK9YwOG1aqD3RYr1JHaJOXUkopt9AailJKKbfQGopSSim30ICilFLKLTSgKKWUcgsNKEoppdxCA4pSSim30ICilFLKLf4fr/IsFRGqSGgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label = 'avg')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label = 'min')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label = 'max')\n",
    "plt.legend(loc='upper left')\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Rewards')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Rendering Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "done = False\n",
    "state = env.reset()\n",
    "while not done:\n",
    "    action = np.argmax(q_table[get_discrete_state(state)])\n",
    "    next_state, _, done, _ = env.step(action)\n",
    "    state = next_state\n",
    "    env.render()\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (spinningup)",
   "language": "python",
   "name": "spinningup"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
